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ABSTRACT 


This  work  Is  concerned  with  the  knowledge  that  electronics 
technicians  possess  of  electronic  equipment,  and 
more  generally,  with  how  people  operate  In  tasks 
that  draw,  upon  a  complex  spatial  symbolic  knowledge 
base.  A  technician's  knowledge  base  is  postulated 
to  consist  of  three  types  of  related  knowledge:  (a) 
structural /functional  knowledge,  which  pertains  to 
the  actual  configuration  of  a  circuit  and  the  role 
that  Its  components  play  in  the  operation  of  the 
device;  (b)  prototypical  knowledge,  which  pertains 
to  the  general  properties  common  to  circuits  of  a 
given  type;  and  (c)  procedural  knowledge,  which  pertains 
to  the  way  that  a  circuit  can  be  modified  and  to 
the  Interaction  among  knowledge  elements  of  all  three 
types  of  knowledge.  A  previous  report  documented 
a  study  of  Individual  differences  In  structural  knowledge; 
the  present  report  focuses  on  an  experiment  conducted 
to  Investigate  Individual  differences  In  procedural 
knowledge.  Novice  and  expert  subjects  performed 
tasks  in  which  they  had  to  either  locate  and  correct 
an  error  In  a  circuit,  change  the  function  of  a  circuit, 
or  complete  a  missing  segnent  In  a  circuit.  On  all 
tasks,  experts  were  found  to.  be  far  more  accurate 
than  novices;  but  more  important,  experts  were  classified  — 
on  the  basis  of  verbal  protocols  —  to  be  considerably 
more  systematic,  orderly  and  directed  In  their  problem 
solving  strategies.  The  productive  procedures  used 
by  experts  were  then  translated  Into  specific  guidelines 
toward  Improving  circuit  troubleshooting,  and  the 
effectiveness  of  these  guidelines  will  be  evaluated 
In  a  subsequent  experiment.  The  results  of  this 
research  program  should  help  In  providing  guidelines 
for  training  electronic  techniques  to  better  understand 
and  troubleshoot  complex  equipment. 
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1.  INTRODUCTION 


1.1  Summary 

This  report  describes  the  second  year  of  a  three-year  program  of  research 
devoted  to  understanding  the  knowledge  base  required  of  experienced 
technicians  to  troubleshoot  complex  electronic  equipment.  The  primary 
objectives  of  this  program  are  to  (1)  describe  the  mental  representations 
of  electronic  devices  that  technicians  derive  from  schematic  circuit 
diagrams,  and  (2)  characterize  the  ways  In  which  technicians  apply  these 
mental  representations  to  trouble-shooting  and  problem  solving.  In  the 
first  year,  we  largely  accomplished  the  first  objective  with  a 
macro-experiment  designed  to  assess  the  structural/functional  knowledge  of 
electronic  devices  possessed  by  technicians  varying  In  skill  level. 
Observed  performance  differences  Include  errors  In  characteristic  places  In 
the  circuits  and  global  differences  In  the  structure  and  organization  of 
the  knowledge  base. 

It  was  our  view  that  these  differences,  which  are  primarily  the  knowledge 
of  facts  about  the  circuit,  were  not  sufficient  to  explain  the  differences 
In  skill  between  technicians  nor  are  they  probably  the  most  Interesting  for 
Investigation.  In  particular,  they  do  not  address  the  way  that  the  circuit 
knowledge  Is  used.  One  could  train  novices  on  the  memorization  of  circuit 
diagrams  until  they  made  few  errors  In  retention,  but  they  would  not  become 
experts  through  this  training.  Practical  tasks  require  more  than  static 
comprehension  of  a  device.  The  knowledge  base  Is  valuable  only  to  the 
extent  that  Its  holder  can  operate  upon  It.  It  was  our  hypothesis  that  the 
Increased  proficiency  of  technicians  derived  through  experience  Is  due  to 
the  acquisition  of  more  sophisticated  procedural  knowledge,  not  simply  to 
the  development  of  a  more  complex  static  knowledge  base.  Accordingly,  In 
the  second  year,  we  concentrated  on  the  study  of  procedural  knowledge,  with 
the  Information  gained  during  the  first  year  providing  the  necessary 
background. 


The  third  year  of  work  will  focus  on  an  Integration  and  summation  of  the 
experiments  performed  during  the  first  two  years.  This  effort  will  Include 
an  experiment  toward  the  development  and  validation  of  guidelines  for 
comprehending  circuits,  building  circuits,  and  troubleshooting. 

1.2  Program  Overview 

1.2.1  Objectives.  Major  objectives  of  the  program  Include  the  following: 

(1)  Describe  the  mental  representation  of  electronic  devices  that 
technicians  derive  from  schematic  circuit  diagrams. 

(2)  Characterize  the  procedural  knowledge  that  technicians  apply 
to  the  mental  representation  to  perform  troubleshooting  and 
problem  solving. 

(3)  Characterize  the  knowledge  structures  that  differentiate  among 
technicians  of  different  skill  levels. 

(4)  Validate  the  findings  of  the  descriptive  studies  by  means  of 
hypothetl co-deductlve  experimentation. 

Three-Year  Program.  The  research  tasks  targeted  for  the  three-year  program 
are  as  follows: 

(1)  First-Year:  Exploration  of  structural  and  functional 

knowledge.  Develop  stimulus  material  pool.  Perform  several 
studies  with  electronic  technicians  varying  In  skill  level  to 
assess  the  structural /functional  knowledge  they  possess  of 
electronic  devices  as  represented  by  circuit  diagrams. 
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(2)  Second-Year:  Investigation  of  Procedural  Knowledge.  Conduct 
descriptive  studies  and  hypothesis  testing  experiments  to 
assess  the  procedural  procedural  knowledge  that  Is  brought  to 
bear  on  solving  problems  with  circuit  diagrams  by  both  novice 
and  more  proficient  electronics  technicians. 

(3)  Third-Year:  Integration  and  Summation  of  Studies  Performed 
During  First  Two  Years.  Develop  and  validate  guidelines  for 
comprehending  circuits,  building  circuits,  and  trouble¬ 
shooting.  This  will  Include  an  experiment  suggested  by  the 
findings  from  the  first  two  years. 

1.3  Overview  of  the  Technical  Approach 

An  overview  of  the  technical  approach  for  the  program  is  shown  in  Figure 
1-1.  Work  in  the  first  year  began  with  Task  1,  the  development  of  a  pool 
of  stimulus  circuit  diagram  materials  for  use  in  all  of  the  subsequent 
work.  In  Task  2,  a  macro-experiment  was  conducted  to  reveal  the 
structural /functional  knowledge  possessed  by  electronic  technicians  varying 
In  level  of  expertise.  The  tasks  studied  in  the  first  year  were  a  memory 
task  and  a  component-partitioning  task. 

In  the  second  year.  Task  3  Investigated  the  procedural  knowledge  that 
technicians  apply  to  the  structural/functional  representation  of  electronic 
devices.  Three  separate  circuit-based  tasks  were  studied  In  the  second 
year:  an  error  correction  task,  an  alteration  task,  and  a  completion  task. 


Work  In  the  third  year  will  focus  mainly  on  the  Interplay  between  the 
structural/functional  knowledge  and  the  procedural  knowledge,  but  also  on 
the  development  of  guidelines  for  Interacting  with  circuits. 


FIGURE  1-1. 

OVERVIEW  OF  EXPERIMENTAL  PROGRAM 
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2.  RESEARCH  BACKGROUND 


2.1  Overview 

The  performance  of  complex  casks  such  as  maintaining  mechanical  equipment, 
modifying  and  adapting  computer  programs,  and  troubleshooting  sophisticated 
electronic  devices  requires  highly  skilled  personnel.  All  such  tasks 
require  these  personnel  to  have  a  detailed  understanding  of  the  devices 
that  are  being  repaired  and  maintained.  Thus,  an  understanding  of  how 
people  mentally  represent  complex  devices  is  necessary  to  train  such  per- 
sonnel  and  to  develop  optimal  man-machine  Interfaces  that  such  personnel 
can  use. 


It  is  comonly  accepted  that  the  training  of  personnel  to  troubleshoot 
electronic  circuits  has  been  less  than  successful.  In  a  review  of  the 
status  of  troubleshooting  in  the  military  services.  Bond  and  Towne  (1979) 
state: 


The  main  conclusion  of  this  report, . . (is  that)  trouble¬ 
shooting  of  very  complex  systems  is  difficult  for 
rtmerous  reasons ,  but  the  critical  factor  is  that  the 
technician' 8  cognitive  map  of  essential  physical 
relations  (electronic,  hydraulic ,  electro-mechanical, 
and  so  on)  in  complex  equipment  is  often  incomplete, 
vague,  or  incorrect.  As  long  as  this  is  so,  any  series 
of  checks  and  test  readings,  though  apparently  well 
motivated  and  accomplished ,  cannot  ' close  in'  logically 
on  a  faulty  unit.  (pp.  S-6) 


The  premise  underlying  the  present  program  of  research  is  that  training 
programs  are  less  successful  than  they  could  be  because  they  fail  to  pro¬ 
vide  troubleshooters  with  the  knowledge  required  to  develop  a  sufficiently 
rich  conceptual  structure  of  the  equipment  they  are  working  with.  Accord¬ 
ingly,  in  this  research  we  propose  to  study  the  nature  of  the  knowledge 
base  that  is  necessary  to  repair  and  maintain  complex  electronic  devices. 
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This  chapter  presents  a  rationale  for  the  work  that  will  be  undertaken. 

Its  purpose  is  to  establish  an  underlying  theoretical  framework  for  the 
research  and  to  clarify  our  particular  choice  of  research  direction. 

Section  2.2  discusses  the  knowledge  base  required  for  complex  problem 
solving.  Section  2.3  discusses  the  nature  of  the  mental  representation 
of  complex  equipment  in  relation  to  contemporary  theories  of  long-term 
memory  «*id  the  representation  of  spatial  information.  Section  2.4  dis¬ 
cusses  relevant  research  pertaining  to  the  nature  of  the  mental  represen¬ 
tation  possessed  by  novice  and  expert  problem  solvers.  Section  2.5  presents 
the  theoretical  perspective  that  we  have  adopted  for  the  conduct  of  the 
proposed  research. 

2.2  Knowledge  Base  and  Complex  Problem  Solving 

Troubleshooting  entails  the  Isolation  and  repair  of  malfunctioning  com¬ 
ponents  in  a  device  and,  as  such,  is  a  form  of  problem  solving.  It  may 
be  analysed  in  terms  of  problem  solving  theories  (see,  e.g.,  Greeno,  1978, 
for  a  review).  However,  we  do  not  intend  to  focus  exclusively  on  this 
aspect  of  the  troubleshooting  question.  Presumably,  the  problem-solving 
procedures  operate  on  a  substrate  of  knowledge,  which  includes  a  mental 
representation,  of  a  device.  This  underlying  structure  must  be  clarified 
before  problem-solving  theories  can  be  applied. 

Although  much  attention  has  been  devoted  to  the  study  of  the  procedure 
people  use  in  problem-solving,  less  attention  has  been  paid  to  the  study 
of  the  knowledge  substrate  required  to  solve  complex  problems.  For 
example,  the  commonly  studied  problems  (Tower  of  Hanoi,  Missionaries  and 
Cannibals,  etc.)  require  a  very  limited  information  base.  We  feel  that, 
in  part,  what  characterizes  good  problem  solvers  from  poor  ones  resides 
In  the  knowledge  base  they  bring  to  problem  solving  tasks  rather  than 
mere  differences  in  their  problem  solving  strategies. 
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While  this  position  may  seem  to  be  uncont rovers ial,  it  is  instructive 
to  note  that  much  research  has  been  performed  that  presupposes  a  very 
impoverished  mental  representation  on  the  part  6f  the  problem  solver. 

For  example,  troubleshooting  is  sometimes  viewed  as  a  decision  making 
activity  where  the  troubleshooter's  task  is  to  iteratively  select  and 
test  system  components  to  determine  whether  they  are  faulty  (e.g. ,  Freedy 
&  Crooks,  1975;  Rouse,  1978).  From  this  perspective,  the  troubleshooter 
must  decide  which  system  component,  among  many,  to  test  on  a  given 
iteration.  Some  have  suggested  (e.g..  Bond  &  Towne,  1979;  Rouse,  1978) 
that  an  optimal  strategy  to  use  is  a  procedure  which  eliminates  half  the 
number  of  components  on  each  test  cycle.  To  be  implemented,  a  strategy 
such  as  this  one  merely  requires,  the  troubleshooter  to  have  a  list  of 
system  components  in  his  mind.  Unfortunately,  it  is  not  clear  what  should 
be  on  this  list,  nor  how  it  is  organized.  It  is  here  that  the  real  novice¬ 
expert  differences  lie. 

We  feel  that  to  successfully  repair  complex  equipment,  a  troubleshooter 
requires  several  types  of  knowledge.  First,  factual  knowledge  of  the 
equipment  is  required.  For  example,  troubleshooters  must  know  that  most 
electronic  devices  have  a  power  supply.  Second,  procedural  knowledge  is 
needed.  Sometimes,  such  knowledge  is  quite  specific;  for  example,  "To 
determine  whether  a  capacitor  is  faulty,  take  a  reading  of  the  voltage 
at  point  x.  If  the  voltage  departs  significantly  from  value  y,  the 
capacitor  is  faulty."  Other  procedural  knowledge  is  much  more  general, 
as  in  a  set  of  procedures  for  identifying  what  may  be  wrong  when  no  power 
is  delivered  to  a  device.  Third,  the  troubleshooter  requires  a  mental 
representation  of  the  faulty  device,  of  its  components  and  their  inter¬ 
relationships  to  each  other.  The  basis  for  this  representational  knowledge 
is  provided  by  schematic  drawings  of  the  equipment.  However,  the  avail¬ 
ability  of  schematic  drawings  does  not  mean  that  the  troubleshooter  under¬ 
stands  the  device. 
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Nature  of  the  Mental  Representation 


It  Is  apparent  that  an  expert's  mental  representation  of  a  complex  device 
is  not  isomorphic  to  a  schematic  drawing  of  the  device.  In  fact,  it 
contains  much  more  information.  (See  Brown,  Collins,  &  Harris,  1978  for 
a  similar  point  of  view.)  For  example,  an  expert  readily  can  identify 
functional  units  within  a  circuit  that  are  not  directly  present  in  the 
diagram.  So  the  first  step  is  to  model  the  latent  knowledge  and  the 
cognitive  mechanisms  that  allow  the  expert  to  develop  an  enriched  mental 
representation  from  a  schematic  drawing. 

Contemporary  psychological  theorizing  provides  several  approaches  to 
this  problem.  One  approach  is  to  view  the  mental  representation  of  spatial 
Information  in  terms  of  connected  networks  (Anderson,  1976;  Anderson  & 

Bower,  1973;  Collins  &  Loftus,  1975;  Norman  &  Rumelhart,  1975;  Schank, 

1972).  According  to  such  a  structural  model,  -individual  circuit  components 
are  represented  by  nodes  in  the  network  and  the  interrelationships  among 
components  form  links  between  the  nodes.  Although  representations  of 
this  sort  can  he  constructured  so  as  to  mirror  closely  the  circuit  diagrams, 
they  also  permit  the  use  of  abstract  nodes  to  represent  the  hierarchical 
relationship  among  functional  groupings  of  components.  Most  of  the  work 
with  these  models  has  been  done  in  building  and  testing  theories  of  the 
organization  of  long-term  memory  and  thus  has  been  outside  the  context  of 
problem-solving  tasks.  An  exception  is  the  work  of  Bhaskar  and  Simon 
(1977)  who  have  undertaken  an  analysis  of  the  structure  of  long-term 
memory  used  by  students  solving  problems  in  a  college- level  course  in 
chemical- engineering  thermodynamics. 

A  second  approach  is  to  view  the  internal  representation  as  a  second- 
order  isomorphism  between  an  external  object  and  some  corresponding 
representational  process  with  the  brain  (Shepard,  1978).  According  to 
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this  view,  the  relations  among  imagined  objects  mirror  to  some  extent 
the  functional  relations  among  the  same  objects  as  actually  perceived. 
Studies  based  on  this  view  (e.g..  Cooper,  1975;  Kosslyn,  1975;  Shepard, 
1978)  have  emphasized  the  close  relationship  between  the  physical  nature 
of  material  and  the  chronometrlc  properties  of  the  response.  Because  of 
the  spatial  nature  of  circuit  diagrams.  Isomorphic  representation  such 
as  these  can  plan  an  Important  role  in  any  model. 

A  third  approach  to  the  representation  of  knowledge  Is  procedural.  In 
such  a  view,  what  is  known  is  not  the  static  properties  of  a  circuit 
diagram  but  ways  of  operating  on  it.  A  worker  may  know  how  to  modify 
a  power  supply  to  better  filter  its  output,  with  this  information  taking 
the  form  of  procedures  for  altering  the  circuit  rather  than  a  compendium 
of  facts  about  supply  filters.  Because  of  its  active  representation, 
procedural  knowledge  has  found  most  use  in  simulation  models  (e.g., 
Winograd,  1972)  and  problem  solving  (Newell  &  Simon,  1972).  Some  models 
(e.g.,  Anderson,  1976)  have  incorporated  both  procedural  and  other  forms 
of  representation. 

deRleer  (1979)  and  deRleer  and  Brown  (1980)  have  described,  from  an 
artificial  intelligence  point  of  view,  some  of  the  procedural  strategies 
that  are  required  to  analyze  the  operation  of  a  circuit.  In  particular, 
they  emphasize  the  need  for  multiple  procedural  strategies.  For  example 
deRleer  (1979)  hypothesizes  that  people  use  topological,  functional,  and 
geometric  representations.  In  topological  analysis,  the  topology  of  a 
new  circuit  is  compared  to  that  of  previously  recognized  circuits;  in 
functional  analysis,  the  behavior  of  the  overall  circuit  is  determined 
by  combining  the  behavior  of  the  individual  components;  and  geometric 
analysis  relies  on  the  tacit  graphical  language  engineers  use  when  they 
describe  circuit  topologies  on  paper.  These  representations  are  used  to 
analyze  circuits  in  terms  of  its  "teleology."  Similarly,  Stevens  and 
Collins  (1980)  argue  that  people  maintain  multiple  representations  of 
physical  systems  such  as  of  the  rainfall  process. 


2.4  Comparing  Novices  to  Experts 


The  previous  research  mosc  relevant  to  understanding  how  people  represent 
complex  equipment  from  schematic  drawings  comes  from  research  comparing 
the  performance  of  experts  to  that  of  novices.  Most  research  in  this  area 
has  Indicated  that  experts  differ  from  novices  more  in  perceptual  memorial 
abilities  than  in  logical,  problem-solving  abilities.  If  it  were  simply 
the  case  that  experts  know  most  about  the  task  than  the  novices,  there 
would  be  little  to  be  gained  from  constructing  an  elaborate  representation 
of  the  task.  However,  this  seems  not  to  be  so.  Experts  seem  to  have 
representations  that  differ  qualitatively  from  those  of  novices.  Studies 
supporting  this  position  are  becoming  common. 

Work  by  deGroot  (1966),  Chase  and  Simon  (1973),  and  Simon  and  Chase  (1973), 
comparing  the  performance  of  Master  and  weaker  chess  players,  indicates 
that  Masters  do  not  "see"  ahead  further  than  the  weaker  players.  Instead, 
the  Masters  are  superior  to  weaker  players  in  their  ability  to  perform 
tasks  involving  the  recall  of  actual  chess  positions.  The  superior  per¬ 
formance  of  Masters  in  these  tasks  cannot  be  attributed  to  a  generally 
superior  visual  short-term  memory  capacity  of  the  Masters  because  when 
chess  pieces  are  placed  randomly  on  the  board,  recall  is  equally  poor  for 
Masters  and  weak  players. 

Egan  and  Schwartz  (1979)  demonstrated  that  expert  electronic  trouble¬ 
shooters  have  a  richer  mental  representation  of  circuit  information  than 
novice  troubleshooters.  For  example,  expert  electronic  troubleshooters 
are  better  able  to  remember  circuit  diagram  information  than  are  novice 
troubleshooters,  and  in  reconstruction,  the  experts  recalled  the  diagrams 
in  groupings  of  functional  units.  The  skilled  technician's  advantage 
in  this  task  did  not  hold  for  non-meanlngfully  arranged  symbols. 
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Badre  (1979)  has  found  similar  results  for  the  recall  of  battlefield 
situation  displays.  Military  experts  show  a  marked  advantage  over 
novices  for  plausible  situation  displays  but  not  for  randomly  arranged 
positions.  Furthermore,  military  experts  recall  battlefield  units  on 
the  basis  of  their  functional  relationship  to  each  other. 

These  results  hold  for  other  spatial  tasks  such  as  the  recall  of  GO 
positions  (Reltman,  1976)  and  have  also  been  extended  to  other  non-spatlal 
tasks  such  as  for  the  recall  of  computer  programs  (Reltman,  McKeithen, 

Reuter,  &  Hirtle,  1979)  and  the  solution  of  physics  problems  (Chi,  Feltovich, 
&  Glaser,  1979;  Larkin,  1979;  Larkin,  McDermott,  Simon,  &  Simon,  1980). 

The  most  influential  theoretical  explanation  of  these  data  is  that  experts 
perceive  spatial  stimuli  by  coding  the  stimuli  into  groups  consisting  of 
several  elements  or  chunks.  In  one  version  of  this  theory  (Simon  &  Gil- 
martin,  1973),  the  chunks  have  verbal  labels  that  can  be  retained  in 
short-term  memory  and  decoded  at  the  time  of  recall.  It  is  argued  that 
experts  quickly  represent  an  entire  spatial  representation  in  a  relative¬ 
ly  small  number  of  chunk  labels  and  that  these  labels  can  be  used  to 
reconstruct  the  spatial  representation.  Pauses  between  successively  re¬ 
called  elements,  the  estimated  size  and  number  of  chunks,  and  the  corres¬ 
pondence  of  recall  groupings  in  copying  tasks  support  this  hypothesis. 

The  data  are  in  good  agreement  with  the  semi-hierarchlcal  theory  of  the 
mental  representation  of  complex  equipment  that  we  propose  in  the  follow¬ 
ing  section  (indeed,  were  part  of  the  motivation  for  our  model). 

Moreover,  our  theory  provides  a  mechanism  to  account  for  Egan  and  Schwartz's 
(1969)  observation  that  experts  can  quickly  label  a  circuit  diagram  as 
belonging  to  a  given  class  (e.g.,  "some  sort  of  power  supply"). 

In  sun,  we  do  not  feel  that  the  understanding  of  skilled  performance  in 
tasks  such  as  circuit  analysis  is  possible  without  considering  multiple 
domains  of  knowledge.  All  are  necessary  to  explain  the  richness  of  an 
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expert  troubleshooter's  mental  constructs*  and  our  ultimate  model  must 
Incorporate  elements  of  each.  In  particular,  we  believe  that  the  differ¬ 
ence  between  experts  and  novices  lies  as  much  or  more  in  the  ability  of 
experts  to  draw  from  a  larger  collection  of  operations  —  or  procedural 
knowledge —  than  in  an  understanding  of  the  structural  and  functional 
nature  of  the  parts  of  the  task  or  in  the  reference  to  prototypes.  Much 
previous  work  has  Ignored  the  dynamic,  procedural  aspects  in  favor  of  more 

static  conceptions  of  the  tasks.  The  following  section  presents  our 

0 

theoretical  position  in  more  detail. 

2.5  Theoretical  Perspective 

As  one  reviews  the  literature  on  the  mental  representation  of  tasks  or 
stimuli,  one  is  struck  with  the  extent  to  which  the  derived  representations 
are  well  matched  to  the  task.  This  suggests  to  use  that  the  subject  is 
able  to  adopt  a  mental  representation  that  is  closely  concordant  with  local 
processing  demands.  The  true  mental  representation  must  have  latent  in 
it  a  variety  of  possible  forms  and  structures.  This  observation  dictates 
one  fundamental  principle  behind  the  experiments  that  we  present  here: 
any  analysis  of  the  mental  characteristic  of  a  domain  of  knowledge  must 
derive  from  a  variety  of  tasks  and  must  posit  a  variety  of  individual 
representations. 

tfe  start  by  making  a  distinction  between  two  ways  of  classifying  the 
content  of  a  knowledge  structure.  The  first  refers  to  the  substantive 
content  of  the  representation,  to  what  facts  it  describes;  the  second 
to  the  way  in  which  the  information  relates  to  the  subjects'  knowledge 
base.  Both  of  these  viewpoints  can  be  further  subdivided  as  is  discussed 
below.  We  emphasize  that  this  is  not  a  dichotomy  in  the  knowledge  Itself  — 
a  particular  piece  of  Information  does  not  belong  to  one  or  the  other  — 
but  rather  two  aspects  of  the  same  mental  structure,  both  of  which  must 
be  treated  in  order  to  fully  comprehend  the  information  processing 
abilities  of  a  subject. 


Content .  The  content  of  the  mental  representation  Is  composed  of  a 
number  of  interrelated  and  overlapping  structures.  These  structures  may 
be  hierarchical  in  form  (although  this  is  not  crucial).  The  hierarchical 
character  arises-  from  the  tendency  to  view  portions  of  the  device  as 
units,  without  examining  their  fine  structure  unless  necessary.  For 
example,  one  may  think  of  a  logic-circuit  component  as  a  flip-flop, 
without  analyzing  it  further  unless  forced  to  do  so.  This  hierarchical 
tendency  has  been  supported  in  modern  integrated  circuits  by  the  physical 
modularization  of  rather  complex  functions. in  single  chips  or  modules;  in 
computers,  for  example,  CPUs,  interface  units  (e.g. ,  DARTs),  and  fairly 
extensive  memory  drivers  may  appear  as  single  units. 

The  overlapping  nature  of  the  representation  comes  from  the  fact  that  a 
particular  component  participates  in  several  organizations  at  once.  In 
some  computer  designs,  for  example,  the  circuit  representing  the  fourth 
bit  of  an  accumulator  may  logically  be  analyzed  as  part  of  that  accumulator , 
or  may  be  considered  as  part  of  the  array  of  fourth  bits  'Tver  a  series  of 
registers  such  as  the  accumulator,  program  converter,  etc.  (It  is  inter¬ 
esting  that  both  forms  of  organization  are  reflected  in  the  physical 
design  of  different  computers.)  As  a  more  prosaic  example,  an  electric 
fuel  pump  in  an  automobile  participates  in  both  the  electrical  and  the  fuel 
system  of  the  car.  Presumably  all  of  these  overlapping  organizations  are 
accessible  to  the  subject  which  is  employed  at  a  particular  time  depending 
on  the  dictates  of  the  task.  The  processing  demands  that  the  subject 
places  on  the  mental  representation  enables  one  form  or  the  other. 

We  feel  that  the  cognitive  representation  of  information  given  in  a  circuit 
diagram  may  be  usefully  described  as  a  set  of  parallel  networks.  In  part, 
these  networks  have  a  hierarchical  structure,  in  that  the  terminal  nodes 
represent  individual  circuit  components  (resistors,  capacitors,  etc.), 
the  intermediate  nodes  represent  either  functional  units  (rectifiers. 
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amplifiers,  etc.),  or  physically  proximal  collections  of  components,  and 
the  highest  nodes  represent  the  total  circuit  (a  power  supply,  an  inverter, 
etc.). 

Nature  of  the  Knowledge.  The  second  scheme  of  organization  relates  to 
the  way  that  the  information  is  used.  Particular  tasks  use  knowledge  in 
different  ways.  Although  the  range  of  possible  uses  probably  forms  a 
complex  and  multidimensional  space,  a  good  case  can  be  made  for  classify¬ 
ing  knowledge  about  a  complicated  mechanism  (such  as  a  circuit  diagram) 
into  three  general  classes: 

(1)  Structural  and  functional  knowledge  deals  with  the  way 
that  a  device  in  constructed  and  the  role  that  its  parts 
play  in  the  operation.  One  may  know,  for  example,  that  a 
transformer  serves  to  change  the  voltage  of  an  AC  supply, 
that  a  particular  combination  of  transltors  acts  as  a  flip- 
flop,  and  so  forth.  Fundamentally ,  this  knowledge  is  static; 
it  describes  the  way  that  the  device  works. 

(2)  Prototypical  Knowledge  relates  one  device  to  more  general 
prototypes.  Devices  are  not  understood  in  isolation,  but 
are  related  to  other  devices.  For  example,  experienced 
technicians  are  able  to  quickly  recognize  that  several  diffi¬ 
cult  circuit  diagrams  represent  the  same  class  of  device. 

This  suggests  a  set  of  procedures  that  force  the  constituent 
elements  of  a  circuit  diagram  into  prototypical  configurations. 
One  portion  of  a  circuit  is  a  rectifier,  another  a  Schmitt 
trigger,  etc.  Two  processes  seem  to  be  involved.  First, 

some  form  of  bottom-up  mechanism  simplifies  the  representa¬ 
tion  of  the  diagram  by  replacing  groups  of  nodes  with  a  sin¬ 
gle  node.  Second,  a  top-down  mechanism  attempts  to  fill  in 


missing  nodes  In  partially  matched  prototypes.  This 
knowledge  is  rarely  In  the  form  of  an  exact  parallel  between 
real  devices,  but  relates  the  operation-  of  any  device  to 
an  abstracted  prototype.  This  prototypical  knowledge  re¬ 
duces  greatly  the  burden  of  the  structural  and  functional 
facts. 

(3)  Procedural  knowledge  gives  ways  to  manipulate  the  device. 
Many  tasks  require  more  than  a  static  comprehension  of  a 
device,  demanding  that  some  modification  be  made  of  some 
operation  performed  on  it.  The  knowledge  necessary  to  do 
this  Is  different  from  the  other  two  types,  embodying  a 
series  of  procedures  (algorithmic  or  heuristic)  for  changing 
the  device.  Procedural  knowledge  Is  the  most  complex  of  the 
three,  and  draws  heavily  on  the  others. 

We  feel  that  training  programs  fall  because  the  mental  representation  is 
unobservable,  and  consequently  It  is  difficult  to  determine  whether  the 
troubleshooter  has  developed  an  adequate  representation.  Moreover,  it  is 
not  clear  how  such  a  representation  is  attained.  However  we  do  know  that 
with  experience,  troubleshooters  become  more  proficient  at  their  job. 
Presumably,  increased  proficiency  derived  through  experience  Is  due  to  a 
more  developed  mental  representation  knowledge  base. 

Our  goal  Is  to  obtain  a  better  understanding  of  the  knowledge  base  that 
skilled  troubleshooters  bring  to  bear  in  complex  equipment.  Of  the  three 
types  of  knowledge  that  we  have  discussed,  procedural  knowledge  is  the 
most  difficult  to  study,  for  it  is  the  most  abstract,  and  depends  on  the 
structural/functional  knowledge  for  its  operation.  But  it  is  also  the 
most  important,  because  knowledge  is  valuable  only  to  the  extent  that  It 
lets  its  possessor  manipulate  equipment.  Thus,  the  first-year  work  on 
structlonal/functional  knowledge  provided  the  necessary  background  to 
understand  procedural  knowledge  in  the  second  year  of  work. 


We  note  that  at  thia  .point  we  have  described  a  general  theoretical  posi¬ 
tion,  but  not  the  specific  models  that  instantiate  It  (for  this  distinction 
see  Wicken8,  1982,  Chapter  I).  Specific  models  for  these  processes 
could  be  formulated  using  a  number  of  conventional  representations  such 
as  production  systems. 


3.  REVIEW  OF  FIRST-YEAR  WORK: 
INVESTIGATION  OF  STRUCTURAL/FUNCTIONAL  KNOWLEDGE 


3.1  Overview 


Reviews  of  the  literature  on  the  mental  representation  of  tasks  or  stimuli 
reveal  that  the  derived  representations  are  surprisingly  well  matched  to 
the  task.  This  suggests  that  a  person  Is  able  to  adopt  a  mental 

representation  that  Is  closely  concordant  with  local  processing  demands. 
The  true  mental  representation  must  have  a  variety  of  possible  forms  and 
structures-that  are  latent  In  that  representation.  Consequently,  any 
analysis  of  the  mental  characteristics  of  a  domain  of  knowledge  must  derive 
from  a  variety  of  tasks  ans  must  posit  a  variety  of  Individual 
representations.  Also,  the  understanding  of  skilled  performance  In  tasks 
such  as  circuit  analysis  requires  consideration  of  multiple  domains  of 
knowledge.  That  Is,  different  domains  are  necessary  to  explain  the 

richness  of  an  expert  troubleshooter's  mental  constructs,  and  a 
comprehensive  model  must  Incorporate  elements  of  each  domain. 

The  experiment  conducted  during  the  first  year  was  designed  to  assess  the 
structural  and  functional  knowledge  of  electronic  devices  possessed  by 
technicians  varying  In  skill  level.  This  work  provided  the  necessary 
background  for  the  Investigation  of  procedural  knowledge  as  related  to 
electronic  trouble-shooting.  The  experiment  performed  was  a  composite  of  a 
circuit-reconstruction  task  and  a  circuit-partitioning  task. 

The  principal  finding  evident  from  the  data  was  the  large  degree  of  varia¬ 
bility  In  the  performance  among  the  subjects  on  the  reconstruction  and 
cluster-generating  tasks,  and  the  extent  of  overlap  between  the  different 
ability  groups  on  those  tasks.  These  data  stand  in  contrast  to  the 
relatively  large  differences  between  skill  levels  that  appeared  In  the  time 
and  error  measures  with  respect  to  overall  performance.  Even  If  a  large 
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sample  could  provide  statistically  significant  differences  between  skill 
levels,  the  size  of  the  effect  as  a  proportion  of  variability  would 
probably  not  be  very  great.  Thus,  the  Investigation  of  structural /func¬ 
tional  knowledge  was  not  the  most  productive  place  for  further  research 
ef f ort . 

Identifiable  differences  in  the  organization  of  a  diagram  during  learning 
and  recall  do  not  appear  to  be  the  most  sensitive  loci  of  proficiency 
differences.  For  reasonably  simple  and  well-learned  material,  the  physical 
properties  of  the  diagram  layout  may  dominate  performance.  It  Is 
conceivable  that  differences  could  appear  between  subjects  of  different 
proficiency  levels,  with  the  less  proficient  subjects  being  more  bound  by 
the  physical  construction  of  the  target  circuit,  and  the  more  proficient 
subjects  being  more  bound  by  the  logical  construction.  In  conventionally 
well-drawn  circuit  diagrams,  however,  these  two  organizations  coincide, 
minimizing  differences  In  performance. 

In  contrast,  more  substantial  differences  are  more  likely  to  appear  In  the 
way  In  which  the  circuit  diagrams  are  manipulated,  that  Is,  In  operations 
that  are  performed  on  the  diagrams.  Even  with  Improperly  drawn  diagrams, 
where  the  logical  and  physical  aspects  conflict,  the  expert's  advantage 
would  come  through  an  ability  to  reorganize  the  circuit.  Thus,  the 
principal  performance  differences  between  the  most  skilled  and  least 
skilled  subjects  may  be  a  result  of  differences  In  their  respective  levels 
of  procedural  knowledge.  For  this  reason,  the  second  year  of  research  work 
was  designed  to  examine  performance  on  tasks  that  require  technicians  to 
manipulate  circuits  In  prescribed  ways.  Such  research,  focusing  on  the 
Investigation  of  procedural  knowledge,  complements  the  exploration  of 
Individual  differences  In  structural/functlonal  knowledge  performed  during 
the  first  year  of  work. 
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4.  REVIEW  OF  SECOND- YEAR  WORK: 
INVESTIGATION  OF  PROCEDURAL  KNOWLEDGE 


4.1  Overview 


Knowing  the  way  In  which  the  mental  representation  of  experts  differ  from 
that  of  novices  does  not  directly  indicate  how  the  experts'  representation 
leads  to  superior  troubleshooting.  Greeno  (1978)  makes  the  same  point  by 
noting  that  while  current  theorizing  gives  an  explanation  of  the  skill  that 
chess  masters  show  In  the  short-term  recall  of  positions,  there  has  been  no 
strong  theoretical  analysis  showing  how  the  existence  of  a  large  store  of 
recognizable  patterns  contributes  to  successful  problem  solving.  Thus, 
tasks  that  tap  procedural  knowledge  are  Intended  to  create  a  bridge  between 
the  structural/functional  nature  of  the  mental  representation,  and  the 
procedures  acting  on  this  representation  that  enable  problem-solving  and 
troubleshooting  activities. 

Three  tasks  were  designed  to  study  procedural  knowledge.  Each  task 
required  the  subject  to  manipulate  a  circuit  diagram  In  some  manner  such 
that  differences  In  algorithmic  or  heuristic  procedures  for  changing  the 
device  can  be  evaluated  as  a  function  of  skill  level.  Our  interest  here 
was  primarily  In  the  interaction  of  proficiency  with  the  specific  diagram 
manipulation.  Task  1  required  the  subject  to  locate  an  error  In  the 
construction  of  a  circuit  on  the  basis  of  symptoms;  Task  2  required  the 
subject  to  change  the  function  of  a  circuit;  and  Task  3  asked  the  subject 
to  complete  a  missing  segment  of  a  circuit.  Each  experiment  is  described 
briefly  below. 
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4.2  Task  1;  Error  Correction  ("FIX"  Task! 

In  this  task,  technicians,  varying  In  skill  level,  were  presented  with  a 
diagram  containing  an  error  In  It  and  with  symptoms  of  its  malfunction. 
The  subject's  task  was  to  find  the  error  and  to  correct  or  fix  it.  While 
performing  this  task,  a  record  was  maintained  of  the  operations  that  the 
subject  used  and  of  their  order,  so  that  the  nature  and  order  of  the 
procedures  could  be  catalogued  and  compared  across  the  specified  stimulus 
material  and  as  a  function  of  subject  skill  level. 

4.3  Task  2:  Change  Function  ("ALTER"  Task) 

A  good  Indication  of  a  subject's  deep-level  comprehension  of  a  circuit  Is 
provided  by  the  readiness  with  which  that  circuit  can  be  modified.  In  this 
experiment,  subjects  were  presented  with  a  circuit  diagram  of  a  device  that 
was  complete  and  would  operate  as  described.  The  subject's  task  was  to 
make  a  specific  change  In  one  of  the  circuit’s  parameters.  For  example.  In 
the  small  diagram  used  In  the  first  year  of  work,  the  subject  might  be 
asked  to  change  the  frequency  of  the  tuning  unit.  As  In  Task  1,  protocols 
of  the  subject's  operations  were  catalogued  and  compared. 

4.4  Task  3:  Complete  Circuit  ("COMPLETE"  Task) 

This  task  required  the  subject  to  fill  In  a  missing  segment  of  a  circuit 
diagram  so  that  the  circuit  would  perform  a  particular  function.  Circuits 
were  constructed  In  which  a  particular  segment  of  the  diagram  was  left 
blank.  These  omissions  subsumed  a  functional  unit  of  the  circuit  (not  just 
a  single  component),  but  were  not  so  large  as  to  require  substantial 
redesign.  As  In  Tasks  1  and  2,  a  record  was  maintained  of  the  procedural 
steps  used  by  the  subjects  to  perform  the  task.  These  protocols  were  then 
summarized  and  compared  between  subject  groups  of  different  skill  level. 
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4.5 


Subjects 

The  sample  of  12  novice  technicians  for  the  second  year  of  work  were  drawn 
from  electronics  trade  schools  In  the  Los  Angeles  area  which  offer  a 
two-year  training  program  in  electronics  to  high  school  graduates.  The  9 
expert  technicians  were  currently  employed  with  2  to  4  years  of  college 
education  and  an  average  of  5  years  of  work  experience  in  electronics. 

4.6  Materials 


Six  circuits  were  designed  for  use  in  the  studies  of  procedural  knowledge. 
Three  of  these  are  analog  circuits  with  a  level  of  complexity  comparable  to 
that  of  the  medium-sized  circuit  used  in  the  first  year's  work.  The 
remaining  three  circuits  were  drawn  from  digital  materials  because  (a)  much 
of  contemporary  electronic  equipment  is  digital,  (b)  much  of  the  current 
technical  training  of  technicians  Is  focused  on  digital  circuits,  and  (c) 
we  wish  to  study  procedural  knowledge  as  it  relates  to  both  types  of 
circuits. 

Three  versions  of  each  dircuit  were  prepared:  a  complete  working  circuit 
for  the  alter  function  task;  an  incomplete  circuit  for  the  circuit 
completion  task;  and  a  modified,  non-working  clrcut  for  the  error 
correctiontask.  These  circuits  are  presented  in  Figures  4-1A  to  4-6C.  In 
each  figure,  the  complete,  working  version  is  shown  in  panel  A,  the 
incomplete  version  In  panel  B,  and  the  modified,  non-working  version  in 
panel  C. 


4.7  General  Procedure 


Three  tasks  were  used  to  study  procedural  knowledge.  Each  task  required 
subjects  to  manipulate  a  circuit  diagram  In  some  manner  such  that 
differences  in  algorithmic  or  heuristic  procedures  for  changing  the  device 
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FIGURE  4-1C. 

ULTRASONIC  RUDENT  ELIMINATOR  (ANALOG):  MOOIFIED,  NON-WORKING  CIRCUIT 
TFOR  "FIX"  TASK! 


FIGURE  4-3A. 

STEREO  PREAMPLIFIER  -  ONE  CHANNEL  (ANALOG):  COMPLETE,  WORKING  CIRCUIT 
[FOR  "ALTER"  TASK] 


FIGURE  4-4A. 

UNIVERSAL  4-BIT  BINARY  COUNTER  (DIGITAL):  COMPLETE,  WORKING  CIRCUIT 
[FOR  "ALTER"  TASK] 


UNIVERSAL  4-BIT  BINARY  COUNTER  (DIGITAL):  INCOMPLETE  CIRCUIT 
[FOR  "COMPLETE"  TASK] 


SHIFT 


FIGURE  4-5A. 

FOUR-BIT  REGISTER  (DIGITAL):  COMPLETE,  WORKING  CIRCUIT 
[FOR  "ALTER"  TASKJ 


FIGURE  4-6A. 

ANALOG  TO  DIGITAL  CONVERTER  (DIGITAL):  COMPLETE ,  WORKING  CIRCUIT 
[FOR  "ALTER"  TASK! 


could  be  evaluated  as  a  function  of  the  subject's  skill  level.  Our 
Interest  here  was  primarily  In  the  Interaction  of  subject  proficiency  with 
the  specific  circuit  manipulation  required  In  the  diagram. 


The  first  task  required  the  subject  to  locate  and  correct  an  error  In  the 
construction  of  a  circuit  on  the  basis  of  symptoms;  the  second  task 
required  the  subject  to  change  the  function  of  a  circuit;  and  the  third 
task  asked  the  subject  to  complete  a  missing  segment  of  a  circuit.  Each 
subject  solved  problems  of  all  three  types  over  the  course  of  one  two-hour 
session.  Each  task  was  performed  twice,  once  with  a  digital  circuit  and 
once  with  an  analog  circuit.  The  selection  of  circuits  for  tasks  was 
counterbalanced  across  subjects  within  each  skill  level;  and  the  order  of 
presentation  of  the  three  different  tasks  was  also  counterbalanced. 


Considerable  effort  was  devoted  to  training  the  subjects  to  produce  the 
verbal  protocols  while  solving  the  task  problems.  The  following 
instructions  were  read  to  the  subjects. 


"We  are  interested  In  your  solutions  to  the  problems,  but  we 
are  also  Interested  In  how  you  arrive  at  your  solutions. 
There  may  be  several  different  ways  to  solve  the  problems  and 
there  are  certainly  several  different  ways  to  arrive  at  any 
one  solution.  We  would  like  to  keep  a  record  of  everything 
you  do  In  coming  up  with  your  solutions.  One  way  that  we  will 
do  this  Is  to  have  you  think  out  loud  as  you  are  solving  the 
problems.  Since  most  people  do  not  naturally  think  out  loud, 
this  may  require  some  concentration  on  your  part. 

First,  it  would  be  most  useful  to  us  If  we  could  write  down  a 
list  of  distinct  steps  that  you  go  through  to  arrive  at  your 
final  solution.  To  help  us  do  this,  we  want  you  to  tell  us 
that  you  are  thinking  at  each  point  as  you  work  toward  your 
solutions,  and  also  why  you  are  think  about  the  steps  you  give 
us. 

It  Is  crucial  to  us  that  you  do  not,  for  any  reason,  edit 
anything  out  from  your  thinking.  For  example,  if  you  see  a 
step  that  you  would  like  to  carry  out  but  you  don't  see  how  to 
do  It  right  away,  please  describe  what  you  would  like  to  do 
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4.8.4  Content  Differences.  Several  reliable  differences  in  the  content 

of  the  protocols  as  a  function  of  skill  level  were  apparent  and  these 
important  differences  form  the  basis  for  procedural  guidelines  to  be 
established  and  evaluated  during  the  third  year  of  work. 

(1)  Experts  devoted  considerable  time  at  the  beginning  of  each 
problem  "learning  how  the  circuit  works"  before  initiating  a 
problem- solution  sequence.  Specifically,  experts  partitioned 
the  circuit  while  labeling  key  parts  from  the  problem 
description  and  traced  the  flow  of  information  through  the 
circuit  while  noting  the  activity  along  the  route.  The 
novices'  Initial  Inspection  time  was  spent  in  unsystematic 
viewing  of  the  circuit. 

(2)  The  second  major  difference  was  In  the  establishment  of  a  plan 
to  attack  the  problem  with  clearly  defined  goals,  and  subjects 
that  link  the  desired  state  of  the  circuit  to  the  current 
state.  The  experts  were  more  patient,  relying  less  on  a 
"shotgun"  approach. 

(3)  The  experts  proceeded  through  a  sequence  of  steps,  making 
changes,  additions,  or  deletions  In  the  circuit,  and  each  step 
was  followed  by  tracing  the  effects  of  the  action  taken  on  the 
functioning  of  the  circuit.  Novices  rarely  studied  the  Impact 
of  each  solution  step  separately,  and  on  occasion,  the  impact 
of  the  entire  solution  was  not  traced  through  the  circuit. 
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4.8.  Results 


4.8.1  Accuracy  of  Solutions.  Accuracy  data  in  terms  of  the  proportion  of 
problems  for  which  workable  solutions  were  provided  for  each  task  are 
presented  In  Table  4-1.  The  data  confirm  that  the  tasks  studied  here  were 
sufficiently  difficult  to  promote  large  accuracy  performance  differences 
with  respect  to  skill  level;  and,  clearly,  the  expert  subjects  did 
significantly  better  than  the  novice  subjects  on  all  tasks.  Furthermore, 
given  that  performance  by  experts  (at  least  on  the  FIX  and  COMPLETE  tasks) 
was  not  perfect.  It  Is  evident  that  the  tasks  required  procedural  knowledge 
and  were  not  identifiable  "textbook"  examples  of  standard  problems. 
Nevertheless,  as  demonstrated  In  previous  stages  of  this  research, 
differences  In  accuracy  performance  are  not  particularly  useful  for 
understanding  procedural  knowledge. 

4.8.2  Classification  of  Protocols.  The  transcribed  protocols  first  were 
Inspected  for  patterns  by  an  experimenter  who  was  blind  to  the  skill-level 
classification  of  each  subject.  All  of  the  protocols  could  be  described  In 
a  general  respect  using  four  classifications  of  problem  solving  behavior: 
directed,  re-directed.  Immediate,  and  trlal-and-error.  Directed  behavior 
Is  characterized  by  an  apparent  plan  toward  solution,  with  goals  and 
if-then  operations.  Re-directed  behavior  has  the  characteristics  of 
directed  behavior,  plus  the  addition  of  at  least  one  decision  on  the  part 
of  the  subject  that  the  current  direction  is  Incorrect  and  a  new  direction 
Is  pursued.  Immediate  solution  behavior  contains  no  sequence  of 
operations,  but  rather  a  "snap"  solution  given  quickly.  Trlal-and-error 
behavior  Is  best  characterized  as  "maybe  I  would  try  X;  no  maybe  I  would 
try  Y." 

This  behavioral  classification  was  carried  out  by  one  of  the  project 
Principal  Investigators  who  was  blind  as  to  which  protocols  were  from 
expert  versus  novice  technicians.  Each  protocol  was  classified 


systematically  based  on  clear  breaks  in  the  solution  paths.  If-then 
statements,  and  admissions  of  incorrect  attempts  and  guessing.  The  results 
are  presented  in  Table  4-2.  As  is  obvious  from  the  data  pattern,  the 
expert  problem  solution  protocols  were  predominantly  classified  (56%)  as 
"directed"  behavior— the  highest  level  of  cognitive  performance.  And,  much 
of  the  balance  of  the  expert  protocols  (28%)  were  clasified  as 
“re-directed"  behavior.  In  contrast  to  the  experts,  the  novices  showed  no 
predominant  behavioral  classification  and  only  33%  of  the  protocols  were 
classified  as  "directed,"  with  nearly  the  same  proportion  being  classified 
as  "trial-and-error." 

The  problem  solving  behavior  classifications  also  revealed  Interesting 
Interactions  between  skill  level  and  the  nature  of  the  task.  Table  4-3 
presents  those  data.  In  particular,  the  most  dramatic  difference  between 
skill  levels  occurs  with  the  ALTER  and  COMPLETE  task;  in  these  tasks,  75% 
of  the  protocols  generated  by  the  experts  were  classified  as  directed, 
whereas  49%  of  the  protocols  generated  by  the  novices  were  classified  as 
trial-and-error.  The  ALTER  and  COMPLETE  tasks  are  task  least  likely  to 
have  received  prior  training  in  the  trade  schools.  It  is  reasonable  that 
these  tasks  represent  exercises  where  the  novices  cannot  mimlck  the 
systematic  behavior  of  experts,  and  therefore  these  type  of  data  may  prove 
most  diagnostic  In  evaluating  differences  between  experts  and  novices  on 
procedural  knowledge. 

4.8.3  Solution  Time.  Table  4-4  presents  the  average  problem  solution 
time  as  a  function  of  skill  level  and  task.  Most  striking  Is  the  lack  of 
differences  In  solution  time  between  skill  levels.  The  only  apparent 
difference  Is  In  the  COMPLETE  task,  where  the  experts  studied  the  diagram 
for  nearly  two  minutes  longer  before  initiating  a  solution  sequence.  This 
Is  consistent  with  the  finding  that  novices  resort  to  trial-and-error 
behavior  on  the  completion  task. 
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TABLE  4-1 


Proportion  of  Problems  for  which 
Workable  Solutions  were  Provided 


FIX  ALTER 

COMPLETE 

EXPERT 

.50  1.00 

.75 

NOVICE 

.  20  .20 

.25 

TABLE  4-2 

Proportion  of  Protocols  Falling 
Into  Each  Classification  As 

A  Function  of  Skill  Level 

DIRECTED  RE-DIRECTED  IMMEDIATE 

TRIAL-AND-ERROR 

DIRECTED  RE-DIRECTED  IMMEDIATE  TRIAL-AND-ERROR 


FIX 


EXPERT 

.67 

.33 

.00 

.00 

NOVICE 

.60 

.30 

.10 

.00 

ALTER 

EXPERT 

.50 

.33 

o 

o 

• 

.17 

NOVICE 

.10 

.40 

o 

o 

• 

.50 

COMPLETE 

EXPERT 

.50 

.17 

.17 

.17 

NOVICE 

.20 

..10 

.30 

.40 

TABLE  4-4 

Average  Solution  Time  (Min) 


FIX  ALTER  COMPLETE 

TOTAL  SOLUTION  TIME: 

EXPERT 
NOVICE 

INITIAL  STUDY  TIME: 


11.9 

13.8 


16.8 

15.9 


14.5 

8.4 


5.  EXPERIMENTAL  PROGRAM  FOR  THIRD  YEAR 


5.1  Overview 

The  first  year  of  work  explored  a  characterization  of  the  knowledge 
structures  that  differentiate  among  technicians  of  different  skill  levels. 
The  second  year  of  work  increased  our  understanding  of  the  procedural 
knowledge  that  technicians  apply  to  the  mental  representation  of  electronic 
circuits  to  perform  troubleshooting  and  problem  solving.  These  latter 
differences,  we  believe,  characterize  the  principal  differences  between 
skilled  and  unskilled  subjects.  The  results  should  then  be  validated  and 
applied. 

It  seems  possible  to  combine  the  validation  of  our  findings  with  some  more 
practical  goals.  Based  on  the  differences  that  we  have  observed,  our  next 
task  can  be  to  develop  and  test  procedural  guidelines  for  comprehending 
circuits,  building  circuits,  and  trouble-shooting.  Our  approach,  then, 
during  the  third  year  of  work  is  to  describe  how  expert  technicians 
approach  the  range  of  tasks  studied  during  the  second  year  of  work,  develop 
specific  procedural  guidelines  based  on  this  knowledge,  and  then  conduct  an 
experiment  to  validate  the  usefulness  of  the  guidelines  toward  Improving 
performance.  The  latter  step  will  be  accomplished  by  providing  one  group 
of  less  proficient  technicians  with  the  guidelines,  and  comparing  their 
performance  to  that  of  a  group  of  technicians  with  comparable  skills  who 
are  not  given  the  guidelines.  This  general  procedure  has  been  employed 
successfully  by  Samet  and  Gelselman  (1981)  to  develop  guidelines  for 
summarizing  tactical  Intelligence  data.  The  basic  notion  Is  that  experts 
have  a  mental  representation  of  their  area  of  expertise  that  is  concordant 
with  the  Information  processing  requirements  that  operations  on  the 
material  entail.  This  representation  can  be  translated  to  some  extent  Into 
guidelines  that  can  be  used  to  improve  the  performance  of  other  individuals 
who  have,  as  yet,  a  more  limited  or  otherwise  less  coherent  mental 
representation  of  these  procedures. 


As  outlined  in  the  discussion  of  the  second  year's  experiments  on 
procedural  knowledge  (see  Section  4.8),  one  striking  difference  between 
experts  and  less-proficient  technicians  is  observed  in  their  initial 
examination  of  the  circuit.  The  experts  spend  a  good  deal  of  time, 
initially,  understanding  the  entire  circuit,  often  without  consideration  of 
the  specific  task  at  hand.  The  less-proficient  technicians,  on  the  other 
hand,  appear  to  enter  the  problem  too  early,  before  they  actually 
comprehend  the  circuit.  There  are  many  reasons  why  this  difference  could 
exist,  but  it  lends  Itself  well  to  the  guidelines  manipulation  outlined 
above. 

It  is  proposed  that  one  study  to  be  carried  out  during  the  third  year  of 
work  be  directed  toward  an  evaluation  of  orienting  tasks  that  would  mlmick 
the  Initial  study  of  the  circuits  exhibited  by  the  experts.  A  general 
framework  for  such  a  study  is  outlined  below.  Other  studies  of  this  type 
could  be  generated  based  on  further  analyses  of  the  data  obtained  from  the 
second  year  research  program. 

5.2  Subjects 

The  sample  of  novice  technicians  for  the  third  year  of  work  will  be  drawn 
from  electronics  trade  schools  in  the  Los  Angeles  area.  A  novice  is 
defined  as  a  student  in  the  early  first  year  of  a  two-year  program.  Each 
subject  will  be  assigned  to  one  of  three  conditions,  based  upon  a  pre-test 
of  structural /functional  knowledge  of  electrical  circuitry.  The  pre-test 
will  insure  that  the  subjects  are  matched  across  conditions  with  respect  to 
this  level  of  structural/functional  knowledge. 
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5.3 


Materials 


The  same  six  circuits  designed  for  the  second  year  of  work  (3  digital,  3 
analog)  and  the  three  tasks  (fix  error,  alter  circuit,  complete  circuit) 
will  be  used  again.  This  will  allow  the  data  from  this  study  to  be 
compared  to  those  generated  by  the  expert  technicians  from  the  second  year 
of  work. 


5.4  Procedure 


Each  subject  will  be  assigned  to  one  of  three  conditions.  In  the  general 
orienting-task  condition,  the  subject  will  be  required  to  answer  certain 
questions  about  the  circuit  at  hand  before  going  on  to  solve  the  problem. 
These  questions  will  serve  to  elicit  Information  sought  by  the  experts 
across  circuits  toward  comprehending  the  circuits  prior  to  problem 
solution.  The  questions  used  in  this  condition  are  considered  general 
because  they  could  be  asked  In  the  context  of  any  circuit  of  the  types 
studied  here.  The  instructions  derived  for  this  condition  are  as  follows: 


Solving  the  type  of  problem  that  we  will  be  giving  you  is  not 
easy  —  many  of  our  subjects  have  difficulty.  To  try  to  make 
the  task  easier,  we  have  some  things  that  you  can  do  to  start 
out.  These  will  help  you  to  understand  the  circuit.  As  you 
start  the  problems,  please  try  to  follow  these  instructions  as 
closely  as  possible. 

The  first  thing  to  do  Is  to  try  to  understand  the  circuit  that 
you  are  working  with.  To  do  this,  you  should  forget  about  the 
problem  you  have  to  solve  for  a  moment  and  concentrate  on  the 
original  diagram.  If  you  do  not  know  first  how  It  works,  you 
are  going  to  be  able  to  fix  or  to  change  It!  As  you  follow  the 
guidelines  below.  It  may  be  helpful  If  you  go  back  and  forth 
from  one  to  another,  since  each  question  helps  to  answer  all 
the  others. 


1 


1 

J 


1.  Start  with  the  purpose  for  the  circuit.  Does  it  have  an 
input?  An  output?  Does  it  do  something  —  perhaps  make  a 
noise.  Are  there  controls  that  one  would  adjust  in  using 
it?  identify  these  Inputs,  outputs,  and  controls. 

2.  Try  to  identify  how  “Information"  flows  through  the 
circuit.  How  does  the  input  get  to  the  output?  Where  is 
It  acted  on  by  the  controls?  Of  course  in  many  circuits 
there  isn't  one  single  path  from  Input  to  output  --  there 
may  be  several  different  connections.  Nevertheless,  it  is 
often  possible  to  trace  one  main  path. 


3.  Try  to  understand  what  the  different  parts  of  the  circuit 
do.  What  are  the  various  components  and  how  do  they  act. 
Two  things  can  be  helpful  here: 

(a)  Do  certain  groups  of  components  go  together  and  act 
as  a  whole?  Several  parts  may  form  a  voltage 
divider,  an  amplifier,  a  counter,  or  the  like.  Try 
to  identify  these  groups  and  to  lable  what  they  do. 
Then  you  can  think  about  them  together,  instead  of 
worrying  about  their  Individual  parts. 

(b)  Are  there  certain  points  in  the  circuit  that  seem  to 
be  particularly  important.  These  might  be  places 
where  several  of  the  groups  you  just  identified 
connect  with  each  other.  They  might  be  places  where 
several  signals  come  together  or  diverge  or  where 
there  is  only  a  single  connection  between  two  parts 
of  the  circuit.  Often  they  lie  on  the  "information" 
paths  that  were  described  in  point  2.  Try  to  decide 
what  the  signal  is  doing  at  these  points. 

4.  What  are  the  dynanic  aspects  of  the  circuit?  Try  to  see 
what  happens  when  the  Input  or  the  controls  are  changed. 
How  does  this  affect  the  output?  How  does  it  affect  the 
important  points  you  identified.  When  you  change  the 
input,  where  does  the  voltage  or  the  current  go  up  or  go 
down? 


Now  turn  to  the  problem  that  you  are  supposed  to  solve.  Start  by  asking 
how  the  changes  you  are  to  make  will  affect  the  parts  of  the  circuit  that 
you  identified.  What  changes  are  necessary?  If  the  circuit  does  not  work 
and  you  are  to  fix  it,  where  might  the  problem  be?  What  changes  are 
necessary?  How  should  the  final  or  complete  circuit  differ  from  what  you 
have  to  start  with? 


1.  Start  with  the  purpose  for  the  circuit.  Does  It  have  an 
input?  An  output?  Does  It  do  something  —  perhaps  make  a 
noise.  Are  there  controls  that  one  would  adjust  In  using 
It?  Identify  these  inputs,  outputs,  and  controls. 

2.  Try  to  identify  how  "Information"  flows  through  the 
circuit.  How  does  the  input  get  to  the  output?  Where  Is 
It  acted  on  by  the  controls?  Of  course  In  many  circuits 
there  Isn't  one  single  path  from  input  to  output  —  there 
may  be  several  different  connections.  Nevertheless,  It  Is 
often  possible  to  trace  one  main  path. 

3.  Try  to  understand  what  the  different  parts  of  the  circuit 
do.  What  are  the  various  components  and  how  do  they  act. 

Two  things  can  be  helpful  here: 

(a)  Do  certain  groups  of  components  go  together  and  act 
as  a  whole?  Several  parts  may  form  a  voltage 
divider,  an  amplifier,  a  counter,  or  the  like.  Try 
to  Identify  these  groups  and  to  lable  what  they  do. 

Then  you  can  think  about  them  together.  Instead  of 
worrying  about  their  Individual  parts. 

(b)  Are  there  certain  points  In  the  circuit  that  seem  to 
be  particularly  Important.  These  might  be  places 
where  several  of  the  groups  you  just  identified 
connect  with  each  other.  They  might  be  places  where 
several  signals  come  together  or  diverge  or  where 
there  Is  only  a  single  connection  between  two  parts 
of  the  circuit.  Often  they  lie  on  the  "information" 
paths  that  were  described  in  point  2.  Try  to  decide 
what  the  signal  Is  doing  at  these  points. 

4.  What  are  the  dynanlc  aspects  of  the  circuit?  Try  to  see 
what  happens  when  the  Input  or  the  controls  are  changed. 

How  does  this  affect  the  output?  How  does  it  affect  the 
important  points  you  identified.  When  you  change  the 
Input,  where  does  the  voltage  or  the  current  go  up  or  go 
down? 

Now  turn  to  the  problem  that  you  are  supposed  to  solve.  Start  by  asking 
how  the  changes  you  are  to  make  will  affect  the  parts  of  the  circuit  that 
you  Identified.  What  changes  are  necessary?  If  the  circuit  does  not  work 
and  you  are  to  fix  it,  where  might  the  problem  be?  What  changes  are 
necessary?  How  should  the  final  or  complete  circuit  differ  from  what  you 
have  to  start  with? 


In  the  specific  orlentlng-task  condition,  the  subject  will  be  required  to 
answer  questions  In  a  similar  manner,  but  these  questions  will  be  specific 
to  the  comprehension  of  the  circuit  at  hand  and  would  not  necessarily  be 
meaningful  for  other  circuits.  In  the  activity  control  condition,  the 
subject  will  be  required  to  answer  questions  about  the  circuit.  An  example 
of  such  a  task  would  be  to  construct  a  parts  list  for  the  circuit,  or  to 
re-draw  the  circuit  layout.  This  manipulation  will  be  Included  to  evaluate 
the  possibility  that  mere  familiarity  with  the  circuit  parts  and  layout 
leads  to  enhanced  performance. 

5.5  Predictions 

It  Is  anticipated  that  the  general  and  specific  orienting  tasks  will 
Influence  the  nature  of  and  resultant  quality  of  the  problem-solution 
protocols  generated  by  the  novice  technicians.  The  protocols  should  become 
more  systematic,  such  as  those  produced  by  the  experts  in  the  second  year 
of  work,  with  less  of  the  "trlal-and-error"  behavior  that  should  again  be 
exhibited  by  the  subjects  In  the  control  condition.  It  Is  hoped  that  the 
general  orienting  tasks  will  be  nearly  as  effective  as  the  specific  ones  In 
Improving  the  quality  of  performance.  The  derivation  of  guidelines  for  the 
comprehension  of  circuit  diagrams  would  be  more  manageable  If  such  a 
pattern  of  results  were  observed. 
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